.\"
.\" Copyright (C) 1994-2021 Altair Engineering, Inc.
.\" For more information, contact Altair at www.altair.com.
.\"
.\" This file is part of both the OpenPBS software ("OpenPBS")
.\" and the PBS Professional ("PBS Pro") software.
.\"
.\" Open Source License Information:
.\"
.\" OpenPBS is free software. You can redistribute it and/or modify it under
.\" the terms of the GNU Affero General Public License as published by the
.\" Free Software Foundation, either version 3 of the License, or (at your
.\" option) any later version.
.\"
.\" OpenPBS is distributed in the hope that it will be useful, but WITHOUT
.\" ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
.\" FITNESS FOR A PARTICULAR PURPOSE.  See the GNU Affero General Public
.\" License for more details.
.\"
.\" You should have received a copy of the GNU Affero General Public License
.\" along with this program.  If not, see <http://www.gnu.org/licenses/>.
.\"
.\" Commercial License Information:
.\"
.\" PBS Pro is commercially licensed software that shares a common core with
.\" the OpenPBS software.  For a copy of the commercial license terms and
.\" conditions, go to: (http://www.pbspro.com/agreement.html) or contact the
.\" Altair Legal Department.
.\"
.\" Altair's dual-license business model allows companies, individuals, and
.\" organizations to create proprietary derivative works of OpenPBS and
.\" distribute them - whether embedded or bundled with other software -
.\" under a commercial license agreement.
.\"
.\" Use of Altair's trademarks, including but not limited to "PBS™",
.\" "OpenPBS®", "PBS Professional®", and "PBS Pro™" and Altair's logos is
.\" subject to Altair's trademark licensing policies.
.\"
.TH pbs_snapshot 8B "20 September 2019" Local "PBS Professional"
.SH NAME
.B pbs_snapshot
- Linux only.  Capture PBS data to be used for diagnostics
.SH SYNOPSIS
.B pbs_snapshot
-h, --help
.br
.B pbs_snapshot 
-o <output directory path> 
.RS 12
[--accounting-logs=<number of days>] 
.br
[--additional-hosts=<hostname list>] [--basic]
.br
[--daemon-logs=<number of days>] 
[-H <server host>]
.br
[-l <log level>] 
[--map=<file path>]
[--obfuscate]
.br
[--with-sudo]
.RE
.B pbs_snapshot
--version
.SH DESCRIPTION

You use 
.B pbs_snapshot 
to capture PBS data for diagnostics.  This tool is written in Python
and uses PTL libraries, including PBSSnapUtils, to extract the data.
You can optionally anonymize the PBS data.  The 
.B pbs_snapshot 
command captures data from all multischeds.  The
command detects which daemon or daemons are running on the host where
it is collecting information, and captures daemon and system data
accordingly.  If no PBS daemons are running, the command collects
system information.  The output tarball contains information about the
host specified via the 
.I -H 
option, or if that is not specified, the
local host.  If you specify additional hosts, the command creates a
tarball for each additional host and includes it as a sub-tarball in
the output.

If you want to capture only PBS configuration information, use the 
.I --basic
option.

.B Required Privilege
.br
The 
.B pbs_snapshot 
command allows you to use the 
.B sudo 
infrastructure
provided by the PTL framework to capture root-owned information via
.I --with-sudo.  
All other information is collected as a normal user.  If
you need to run 
.B pbs_snapshot 
as a non-prvileged user, and without
using the PTL 
.I --with-sudo 
infrastructure, you must be root if you want
root-owned information to be collected.

.B Restrictions
.br
The 
.B pbs_snapshot 
command is not available on Windows.

.SH OPTIONS
.IP "--accounting-logs=<number of days>" 5
Specifies number of days of accounting logs to be collected; this
count includes the current day.
.br
Value of 
.I number of days 
must be >=0:
.br
   If number of days is 0, no logs are captured.
.br
   If number of days is 1, only the logs for the current day 
.br
   are captured.
.br
Default: 
.B pbs_snapshot
collects 30 days of accounting logs

.IP "--additional-hosts=<hostname list>" 5
Specifies that 
.B pbs_snapshot
should gather data from the specified list of additional hosts.  
Launches the 
.B pbs_snapshot 
command on each specified host, creates a
tarball there named 
.I <hostname>_snapshot.tgz, 
and includes it as a sub-tarball in the output for the main output.  If you use the
.I --with-sudo 
option, each launched copy uses that option as well.

The command does not query the server when it runs at a non-server host.  

The command collects a full snapshot, including the following information:
.RS 8
Daemon logs, for the number of days of logs being captured, specified
via the 
.I --daemon-logs=<number of days> 
option
.br
The PBS_HOME/<daemon>_priv directory
.br
Accounting logs if server daemon runs on host
.br
System information
.RE
.IP
Format for 
.I hostname list 
is a comma-separated list of one or more hostnames: 
.br
.I <hostname>[, <hostname> ...]

.IP "--basic" 5
Captures PBS configuration information only.  Captures the following:
.nf

Output File                   Description of Captured Information
------------------------------------------------------------------
server/qstat_Bf.out           Output of qstat -Bf
server/qstat_Qf.out           Output of qstat -Qf
scheduler/qmgr_lsched.out     Output of qmgr -c 'list sched'
node/pbsnodes_va.out          Output of pbsnodes -va
reservation/pbs_rstat_f.out   Output of pbs_rstat -f
job/qstat_f.out               Output of qstat -f
hook/qmgr_lpbshook.out        Output of qmgr -c 'list pbshook'
sched_priv/ for each          Copy of each scheduler's 
scheduler instance            sched_priv directory
server_priv/resourcedef       Copy of server_priv/resourcedef file  
pbs.conf                      Copy of /etc/pbs.conf on server host    
pbs_snapshot.log              Log of pbs_snapshot execution
ctime                         Timestamp of when the snapshot
                              was taken
.fi

Can be combined with other options such as 
.I --accounting-logs
and 
.I --daemon-logs
in order to capture additional information.

.IP "--daemon-logs=<number of days>" 5
Specifies number of days of daemon logs to be collected; this count
includes the current day. 
.br
Value of 
.I number of days 
must be >=0:
.RS 8
If number of days is 0, no logs are captured.
.br
If number of days is 1, only the logs for the current day are captured.
.RE
.IP
Default: 
.B pbs_snapshot
collects 5 days of daemon logs

.IP "-h, --help" 5
Prints usage and exits.

.IP "-H <hostname>" 5
Specifies hostname for host whose retrieved data is to be at the top
level in the output tarball.  If not specified, 
.B pbs_snapshot 
puts data for the local host at the top level in the output tarball.

.IP "-l <log level>" 5
Specifies level at which 
.B pbs_snapshot
writes its log.  The log file is 
pbs_snapshot.log, in the output directory path specified using the 
.I -o <output directory path> 
option.

Valid values, from most comprehensive to least: DEBUG2, DEBUG,
INFOCLI2, INFOCLI, INFO, WARNING, ERROR, FATAL
.br
Default: INFOCLI2


.IP "--map=<file path>" 5
Specifies path for file containing obfuscation map, which is a
<key>:<value> pair-mapping of obfuscated data.  Path can be absolute
or relative to current working directory.
.br
Default: 
.B pbs_snapshot
writes its obfuscation map in a file called obfuscate.map in the
location specified via the
.I -o <output directory path> 
option.  
.br
Can only be used with the 
.I --obfuscate 
option.

.IP "--obfuscate" 5
Obfuscates (anonymizes) or deletes sensitive PBS data captured by 
.B pbs_snapshot.
.br
Obfuscates the following data: 
.RS 8
euser, egroup, project, Account_Name, operators, managers, group_list,
Mail_Users, User_List, server_host, acl_groups, acl_users,
acl_resv_groups, acl_resv_users, sched_host, acl_resv_hosts,
acl_hosts, Job_Owner, exec_host, Host, Mom, resources_available.host,
resources_available.vnode
.RE
.IP " " 5
Deletes the following data: 
.RS 8
Variable_List, Error_Path, Output_Path, mail_from, Mail_Points,
Job_Name, jobdir, Submit_arguments, Shell_Path_List
.RE

.IP "--version" 5
The 
.B pbs_snapshot
command prints its PBS version information and exits.
Can only be used alone.

.IP "--with-sudo" 5
Uses the PTL 
.B sudo 
infrastructure in order capture root-owned
information via 
.B sudo.  
(Information not owned by root is captured
using normal privilege, not root privilege.)  With this option, you do
not need to prefix your 
.B pbs_snapshot 
command with 
.B sudo, 
and you do not need root privilege.

.SH Arguments to pbs_snapshot
.IP "-o <output directory path>" 5
Path to directory where 
.B pbs_snapshot
writes its output tarball.  Required.  Path can be absolute or
relative to current working directory.
.br
For example, if you specify 
.I -o /temp,
.B pbs_snapshot
writes "/temp/snapshot_<timestamp>.tgz".
.br
The output directory path must already exist. 

.SH Output
.B Output Location
.br
You must use the 
.I -o <output directory path> 
option to specify the directory where 
.B pbs_snapshot
writes its output.  The path can be absolute or relative to current
working directory.  The output directory must already exist.  As an
example, if you specify "-o /temp", 
.B pbs_snapshot
writes "/temp/snapshot_<timestamp>.tgz".

.B Output Contents
.br
The 
.B pbs_snapshot
command writes 
the output for the local host and each specified remote host as a
tarball.  Tarballs for remote hosts are included in the main tarball.

The command captures JSON output from 
.B qstat-f -F json 
and 
.B pbsnodes -av -F json.  
.br
The main tarball contains the following directory structure, files, and tarballs:
.nf
Directory  Directory
or File    Contents             Description
------------------------------------------------------------------------
server/         
           qstat_B.out          Output of qstat -B
           qstat_Bf.out         Output of qstat -Bf
           qmgr_ps.out          Output of qmgr print server
           qstat_Q.out          Output of qstat -Q
           qstat_Qf.out         Output of qstat -Qf
           qmgr_pr.out          Output of qmgr print resource

server_priv/                    Copy of the PBS_HOME/server_priv 
                                directory.   
                                Core files are captured separately; 
.fi
.RS 32
see
.I core_file_bt/.  
.RE
.nf

           accounting/          Accounting logs from 
                                PBS_HOME/server_priv/accounting/ 
                                directory for the number of days 
.fi
.RS 32
specified via 
.I --accounting-logs 
option
.RE
.nf

server_logs/                    Server logs from the 
                                PBS_HOME/server_logs directory for the
                                number of days specified 
.fi
.RS 32
via
.I --daemon-logs 
option
.RE
.nf

job/            
           qstat.out            Output of qstat
           qstat_f.out          Output of qstat -f
           qstat_f_F_json.out   Output of qstat -f -F json
           qstat_t.out          Output of qstat -t
           qstat_tf.out         Output of qstat -tf
           qstat_x.out          Output of qstat -x
           qstat_xf.out         Output of qstat -xf
           qstat_ns.out         Output of qstat -ns
           qstat_fx_F_dsv.out   Output of qstat -fx -F dsv
           qstat_f_F_dsv.out    Output of qstat -f -F dsv
node/           
           pbsnodes_va.out      Output of pbsnodes -va
           pbsnodes_a.out       Output of pbsnodes -a
           pbsnodes_avSj.out    Output of pbsnodes -avSj
           pbsnodes_aSj.out     Output of pbsnodes -aSj
           pbsnodes_avS.out     Output of pbsnodes -avS
           pbsnodes_aS.out      Output of pbsnodes -aS
           pbsnodes_aFdsv.out   Output of pbsnodes -aF dsv
           pbsnodes_avFdsv.out  Output of pbsnodes -avF dsv
           pbsnodes_avFjson.out Output of pbsnodes -avF json
           qmgr_pn_default.out  Output of qmgr print node @default

mom_priv/                       Copy of the PBS_HOME/mom_priv 
                                directory.
                                Core files are captured separately; 
                                see core_file_bt/.  

mom_logs/                       MoM logs from the PBS_HOME/mom_logs 
                                directory for the number of days 
.fi
.RS 32
via
.I --daemon-logs 
option
.RE
.nf

comm_logs/                      Comm logs from the PBS_HOME/comm_logs 
                                directory for the number of days 
.fi
.RS 32
specified via 
.I --daemon-logs 
option
.RE
.nf  

sched_priv/                     Copy of the PBS_HOME/sched_priv 
                                directory, with all files.
                                Core files are not captured; 
                                see core_file_bt/. 

sched_logs/                     Scheduler logs from the 
                                PBS_HOME/sched_logs directory for 
                                the number of days specified 
.fi
.RS 32
via
.I --daemon-logs 
option    
.RE
.nf

sched_priv_<multisched name>/   Copy of the 
                                PBS_HOME/sched_priv_<multisched_name>
                                directory, with all files.
                                Core files are not captured; 
                                see core_file_bt/. 

sched_logs_<multisched name>/   Scheduler logs from the 
                                PBS_HOME/sched_logs_<multisched_name> 
                                directory for the number
                                of days specified
.fi
.RS 32
via
.I --daemon-logs 
option    
.RE
.nf

reservation/            
           pbs_rstat_f.out      Output of pbs_rstat -f

           pbs_rstat.out        Output of pbs_rstat

scheduler/              
           qmgr_lsched.out      Output of qmgr list sched

hook/           
           qmgr_ph_default.out  Output of qmgr print hook @default

           qmgr_lpbshook.out    Output of qmgr list pbshook

datastore/              
           pg_log/              Copy of the 
                                PBS_HOME/datastore/pg_log directory 
                                for the number of days specified 
.fi
.RS 32
via
.I --daemon-logs 
option
.RE
.nf

core_file_bt/                   Stack backtrace from core files 

           sched_priv/          Files containing the output of thread 
                                apply all backtrace full on all core 
                                files captured from PBS_HOME/sched_priv

           sched_priv_          Files containing the output of thread 
           <multisched name>/   apply all backtrace full on all core 
                                files captured from PBS_HOME/sched_priv

           server_priv/         Files containing the output of thread 
                                apply all backtrace full on all core 
                                files captured from 
                                PBS_HOME/server_priv

           mom_priv/            Files containing the output of thread 
                                apply all backtrace full on all core 
                                files captured from PBS_HOME/mom_priv

           misc/                Files containing the output of thread 
                                apply all backtrace full on any other 
                                core files found inside PBS_HOME

system/         
           pbs_probe_v.out      Output of pbs_probe -v

           pbs_hostn_v.out      Output of pbs_hostn -v $(hostname)

           pbs_environment      Copy of PBS_HOME/pbs_environment file

           os_info              Information about the OS

           process_info         List of processes running on the system 
                                when the snapshot was taken.  Output of
                                ps -aux | grep [p]bs on Linux systems,
                                or tasklist /v on Windows systems

           ps_leaf.out          Output of ps -leaf.  Linux only.

           lsof_pbs.out         Output of lsof | grep [p]bs.
                                Linux only.
           etc_hosts            Copy of /etc/hosts file.  Linux only.

           etc_nsswitch_conf    Copy of /etc/nsswitch.conf file.
                                Linux only.

           vmstat.out           Output of the command vmstat.  
                                Linux only.

           df_h.out             Output of the command df -h.  
                                Linux only.

           dmesg.out            Output of the dmesg command.  
                                Linux only.

pbs.conf                        Copy of the pbs.conf file on the 
                                server host    

ctime                           Contains the time in seconds since 
                                epoch when the snapshot was taken    

pbs_snapshot.log                Log messages written by pbs_snapshot
    
<remote hostname>.tgz           Tarball of output from running the 
                                pbs_snapshot command at a remote host
.fi

.SH Examples
.IP "pbs_snapshot -o /tmp" 5
Writes a snapshot to /temp/snapshot_<timestamp>.tgz that includes 30
days of accounting logs and 5 days of daemon logs from the server
host.

.IP "pbs_snapshot --daemon-logs=1 --accounting-logs=1 -o /tmp --obfuscate --map=mapfile.txt" 5
Writes a snapshot to /temp/snapshot_<timestamp>.tgz that includes 1
day of accounting and daemon logs.  Obfuscates the data and stores the
data mapping in the map file named "mapfile.txt".

